fix(dispatcher): improve fanout delivery timeouts and retry budget#3270
fix(dispatcher): improve fanout delivery timeouts and retry budget#3270migmartri wants to merge 3 commits into
Conversation
- Bump MaxElapsedTime from 10s to 5m (~12-15 retry attempts) - Add per-attempt context.WithTimeout (10s) so a single hung HTTP request cannot consume the entire retry budget - Set http.Client.Timeout on the webhook plugin to match the per-attempt deadline (shared via sdk.PerAttemptTimeout) - Wrap backoff with backoff.WithContext so the retry loop stops promptly when the parent context is cancelled - Replace deprecated io/ioutil with io in the webhook plugin Refs chainloop-dev#3269 Closes chainloop-dev#39 Assisted-by: OpenCode Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev> Chainloop-Trace-Sessions: ses_0c4100537ffeM74j5m1U4O10fe
AI Session Analysis
|
| Status | Attribution | File | Lines |
|---|---|---|---|
| modified | ai | AI_POLICY.md |
+5 / -0 |
| modified | ai | CLAUDE.md |
+3 / -0 |
Policies (4)
| Status | Policy | Material | Messages |
|---|---|---|---|
| ✅ Passed | ai-config-ai-agents-allowed |
ai-coding-session-ses-0c |
- |
| ✅ Passed | ai-config-no-dangerous-commands |
ai-coding-session-ses-0c |
- |
| ✅ Passed | ai-config-no-secrets |
ai-coding-session-ses-0c |
- |
| ✅ Passed | ai-config-mcp-servers-allowed |
ai-coding-session-ses-0c |
- |
Powered by Chainloop and Chainloop Trace
Reflects the new per-attempt HTTP timeout behavior introduced in the previous commit. Assisted-by: OpenCode Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev> Chainloop-Trace-Sessions: ses_0c4100537ffeM74j5m1U4O10fe
migmartri
force-pushed
the
dw/5173-1783416967
branch
from
00441fb to
e16b9b2
Compare
Add OpenCode to the examples and state explicitly that the value must match the tool that produced the work, not the underlying model, and must not be copied blindly from an example. Assisted-by: OpenCode Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev> Chainloop-Trace-Sessions: ses_0c4100537ffeM74j5m1U4O10fe
waveywaves
left a comment
waveywaves
left a comment
There was a problem hiding this comment.
A few inline questions focused on keeping the fix tight and matching existing timeout patterns.
| // set their http.Client.Timeout to this value so a hung endpoint cannot | ||
| // consume the entire retry budget in a single attempt — the cutoff is | ||
| // enforced at whichever layer sees the stall first. | ||
| const PerAttemptTimeout = 10 * time.Second |
There was a problem hiding this comment.
Can this stay out of the SDK? This makes control-plane retry policy part of the plugin SDK API. A local dispatcher/webhook timeout constant is boring duplication, but avoids coupling external plugin authors to this value.
| logger.Infow("msg", "executing integration", "integration", plugin.String(), "input", inputType) | ||
|
|
||
| err := plugin.Execute(ctx, opts) | ||
| err := plugin.Execute(attemptCtx, opts) |
There was a problem hiding this comment.
This only bounds plugins that honor the context. webhook does, but dependency-track currently ignores ctx and uses http.DefaultClient without a timeout, so this isn't a universal fanout attempt timeout yet. Should we update that plugin too or narrow the claim?
| return err | ||
| }, | ||
| b, | ||
| backoff.WithContext(b, ctx), |
There was a problem hiding this comment.
This is good, but the production caller currently uses dispatcher.Run(context.TODO(), ...), so cancellation won't fire there. Should the caller pass a real parent/app context, or should we avoid claiming production cancellation is fixed?
2 participants
Changes
MaxElapsedTimefrom 10s to 5m: ~12-15 retry attempts with the existing exponential backoff, enough to ride out transient blips without leaking goroutines in the fire-and-forget dispatch model.Executecall now runs undercontext.WithTimeoutso a single hung HTTP request cannot consume the entire retry budget.http.Client.Timeouton the webhook plugin: matches the dispatcher's per-attempt deadline via a sharedsdk.PerAttemptTimeoutconstant.backoff.WithContextensures the retry loop stops promptly on parent context cancellation.io/ioutilwithioin the webhook plugin.Closes #3269
Refs #39
Assisted-by: OpenCode
🤖 Posted by Maximus bot (OpenCode) on behalf of @migmartri